Rebased onto master and added a commit with a few comments.

Codecov Report

Merging #179 (1c1be29) into master (b1ee64a) will decrease coverage by 3.15%.
The diff coverage is 81.15%.

❗ Current head 1c1be29 differs from pull request most recent head 11b833e. Consider uploading reports for the commit 11b833e to get more accurate results

@@            Coverage Diff             @@
##           master     #179      +/-   ##
==========================================
- Coverage   78.60%   75.45%   -3.16%     
==========================================
  Files          43       37       -6     
  Lines       18454    17240    -1214     
==========================================
- Hits        14506    13008    -1498     
- Misses       3948     4232     +284     

Rebased onto master and again added a few comments. I'm gonna look into some profiles now.

I profiled a RecogniseGroup call using the group g from tst/working/quick/MatSn.tst. It looks like, without considering time spent in children, the functions SlotUsagePattern, ResultOfLineOfStraightLineProgram, and ResultOfStraightLineProgram contribute most of the overhead.

Maybe, if we rewrote those functions in C or in Julia, we would get rid of most of the overhead.

For the profiling runs and a comparison of the time spent per functions, see the funcoverview.html in each of the archives with-mandarins.tar.gz and without-mandarins.tar.gz.

Also, BlocksModScalars does not seem to play nice with the mandarins but that's a whole other story.

Continued work a bit and rebased onto master.

I simplified a few parts, but am not 100% sure that all simplifications are correct.

rebased onto master

Closed in favor of #280.

As @fingolfin mentioned in a call a few months ago, this will give us a reliable probability estimate as to whether the kernels were constructed correctly. Thus I'll continue work on this PR.

Squashed and rebased onto master.

Rebased onto #308. That makes it easier for me to extend RecogniseGeneric.

So, there are a few errors in the tests and tests now also take 21 minutes on my machine compared to roughly 13 minutes on the GitHub machines. It might have to do with me not storing the SLPs of the mandarins yet. I'll have a look at some profiles.

I seem to not get errors anymore but instead there might be an infinite loop in tst/working/slow/C3C5.tst.

Also, I had a look at Mandarins in CompositionTree again. They compute SLPs for mandarins in the leaves. For mandarins in inner nodes, they from their SLPs from mandarin SLPs in the children. I think this is the magma line which does the job:

    node`MandarinSLPs :=
	[kernelNodeSLPs[i] * preImagesSLPs[i] : i in [1 .. #node`Mandarins]];

So if I understand correctly they create SLPs for the images, and then take each mandarin divided by a preimage of its image, and then find an SLP for that kernel element. In the end, glue everything together. Afterwards, they test whether the evaluating the SLP yields the original mandarin.

I bet this is a lot quicker than recomputing SLPs all the time, which is what I'm currently doing. First I need to get rid of this infinite loop though.

The infinite loop was resolved by ignoring nodes recognised by BlocksModScalars. ri!.isone is broken for these nodes and that just doesn't work with mandarins.

This leaves three open points:

• how to design a dedicated test for the mandarin code?
• handle TryToEnlargeKernelGeneratingSetAndUpdateSLPsDuringMandarinCrisis returning fail. There we have to discard and re-recognise the image recog node.
• Handle the SLPs more intelligently, namely by storing and glueing them together.

Of course there's also documentation! :D

And I vaguely remember something from an email by Eamonn O'Brien, I might be mixing up something though: if the SLPs in a computation become too long because we generated lots of random elements by product replacement, then it might make sense to reset the product replacement random generator of the group in question so that the SLPs become shorter again. 🤔

I'm working on the SLP business. I expect tests to fail.

I added a bunch more commits. Everything seems to be correct and the slow-down due to mandarins seems to be OK-ish for most tests. When we use mandarins immediate verification never finds incorrect kernels anymore, which makes sense since that's the whole point of the mandarins. So I'll see whether I can speed things up by disabling immediate verification when mandarins are enabled.

There are a few tests which become more than two times slower though. In some, mandarin crises get triggered (that is we found out that some kernel was too small) which leads to recomputing half the tree. I think I could improve upon that by being more conservative about which part of the tree gets thrown away during a crisis. If that doesn't resolve the reason for the crisis, we can still chop off more of the tree.

However, there also are a few tests which become super slow, even though no crisis was triggered. I'll have to look into that too.